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SECTION  1 


INTRODUCTION 


Northrop  Corporation,  Electro-Mechanical  Division,  is  pleased  to  subtait  the  first 
quarterly  report  of  the  Experimental  Development  iyf  a FLIR  Sensor  Processor  program. 
The  program's  funding  started  on  July  1,  1976,  and  this  report  summarizes  the  tech- 
nical performance  through  September  30,  1976,  marking  the  completion  of  Task^, 
Experiment  and  Study  Plan,  of  Contract  No.  DAAG53-76-C-0188 . 


Section  2 contains  a brief  Advanced  Scout  Helicopter  mission  scenario  used  to  bound 
the  problem  of  a FLIR  operating  in  a difficult  environment.)  It  includes  a table 
showing  the  effect  of  some  of  the  operational  parameters,xm  image  quality.  Parts  of 
this  section,  which  have  been  classified  been  included  in  a separately 

bound  appendix  along  with  «■  bibliography  of  background  documents. 


Section  3 provides  a basic  definition  of  the  Haar  functions,  since  the  Haar  trans- 
form provides  the  basis  for  all  of  the  image  processing  techniques  presented  in  this 
report.  The  Rationalized  Haar  functions  used  in  these  experiments  are  also  discussed 
in  Patent  No.  3981443,  "Class  of  Transform  Digital  Processors  for  Compression  of 
Muj.frl^lmenslonal  Data." 

^Section  4 describes  a set  of  image  descriptors  designed  to  facilitate  the  evaluation 

of  images  and  image  processing  algorithms  throughout  the  course  of  this  study.,  Using 

/ 

groupings  of  the  Haar  transform  coefficients,  a vector  whose  components  arsL-lfhe 
descriptors  of  Tone,  Edge,  and  Texture  is  defined.  Computer  slnulntlons  using 
stored  photographs  were  utilized  to  develop  these  descriptors.  The  next  phase  of 
this  program,  where  controlled  experiments  will  be  performed  on  test  Imagery,  will 
further  evaluate  the  utility  of  these  image  descriptors. 


Section  5 Includes  the  basic  approach  for  conducting  the  experiments.  Tasks  U 
through  V^of  the  contract.  This  section  briefly  describes  the  Northrop  two- 
dimension  sensor  processor,  ^Ideo  test  equipment  and  the  computer  simulation 
test  setups.  It  then  proceed^  with  the  objectives,  conceptual  solutions. 


I 

i 


and  generalized  experimental  approach  to  principal  areas  In  which  Haar  transform 
processing  can  Improve  FLIR  imaging  and  operating  characteristics.  Specific  areas 
covered  Include  autofocus,  autodeblur,  Image  enhancement,  adaptive  signal  processing 
and  data  compression. 

During  this  reporting  period,  the  two-dimensional  sensor  processor  developed  during 
Northrop' s CY  75  and  CY  76  Independent  Research  and  Development  program,  video  and 
test  equipment  (as  shown  In  Figure  5-1  which  follows)  have  been  completed.  Programs 

•I 

have  been  written  to  interface  the  minicomputer  with  the  sensor  processor,  the  mag- 
netic tape  unit,  high-speed  printer,  and  teletypewriter.  This  has  also  been  a learn- 
ing period  where  the  effects  of  the  many  hardware-implemented  Image  processing  algo- 
rithms designed  Into  the  sensor  processor  have  been  observed,  using  live  video  Inputs. 
As  a result,  several  minor  modifications  have  been  Incorporated  Into  the  processor 
and  test  fixtures  to  Improve  their  operating  capabilities.  We  are  looking  forward 
to  the  next  quarter  of  this  program  when  we  will  begin  to  perform  quantitative  experi- 
ments on  controlled  Imagery. 
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SECTION  2 

MISSION/SYSTEMS  ANALYSIS 


This  section  briefly  describes  the  Advanced  Scout  Helicopter  mission,  and  details 
all  system  parameters  and  visionics  on  board  the  Advanced  Scout  Helicopter  (ASH)/ 
Advanced  Attack  Helicopter  (AAH)  that  affect  image  quality.  The  analysis  of  these 
parameters  and  visionic  specifications  will  bound  the  problem,  to  assure  the  valid- 
ity of  the  experimental  and  study  approaches  that  are  outlined  in  Section  5. 

2.1  ASH  MISSION  SCENARIO 

2.1.1  ASH/AAH  TADS  Description 

An  experimental  FLIR  sensor  processor  is  being  developed  in  this  phase  of  the  con- 
tract for  an  ASH  Target  Acquisition  Designation  System  (TADS)  visionics  application. 
This  FLIR  sensor  processor  could  also  be  used  in  the  AAH  TADS  Visionics  System, 
because  there  is  a good  possibility  that  the  ASH  and  AAH  will  have  common  visionics 
systems.  ASH/AAH  TADS  will  be  used  for  acquiring,  tracking,  ranging,  designating 
and  engaging  tank-size  targets  under  day/night  conditions.  It  will  be  used  in  the 
following  modes: 

a.  Laser  Detection  Mode  to  detect  designated  targets.  The  laser  tracker  will 
operate  in  autotrack,  autosearch,  and  manual  search  modes. 

b.  Laser  Designation  Mode  to  hold  a laser  beam  on  a stationary  or  moving  point 
target  for  handoff  and/or  designation  for  semiactive  terminal  homing  weapons. 

c.  Gun  and  Rocket  Fire  Control  Mode  to  provide  target  range  and  pointing  infor- 
mation for  use  in  gun  and  rocket  weapons  delivery. 

d.  Sight  Cueing  ^tode  to  provide  line-of-sight  cueing  by  other  on-board  sight- 
ing and  navigation  systems. 
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e.  Night  Low  Level  Navigation  Mode  to  be  used  as  a night  pilotage  aid  by 
either  crew  member. 

f.  Target  Acquisition  Mode  to  enable  the  operator  to  visually  search  for  and 
acquire  targets  for  day /night  engagement  at  the  specified  ranges.  Initial  search 
and  target  acquisition  will  be  accomplished  in  a minimum  of  time. 

TADS  is  an  Integrally  mounted  turret  and  consists  of  the  following  subassemblies: 

a.  Turret: 

1.  Day  Viewing  Subsystem  (Direct  View  and  TV) 

2.  Night  Viewing  Subsystem  (FLIR)  and  FLIR  Sensor  Processor 

3.  Laser  Rangefinder/Designator 

4.  Laser  Tracker 

3.  Stabilization  and  Tracking  Subsystem 

b.  Electronic/Signal  Processing 

c.  Optical  Relay. 

2.1.2  AAH  Operational  Modes 

One  of  the  AAH  operational  modes  uses  the  aerial  scout  helicopter  (the  AAH  remote 
aerial  scout  designation  operational  mode).  This  report  concentrates  on  the  mission 
scenario  pertaining  to  this  AAH  operational  mode  using  the  ASH  as  the  aerial  scout 
helicopter. 

In  the  AAH  remote  aerial  scout  designation  (ASH/ AAH)  operational  mode  of  operation, 
ASH  designates  targets  for  the  AAH  with  an  on-board  laser  designator.  ASH  continues 
to  designate  the  target  until  missile  impact,  while  the  AAH  can  remask  immediately 
after  missile  launch.  The  ASH/AAH  helicopter  survivability  and  time  line  analysis 
associated  with  this  operational  mode  are  discussed  in  references  1 and  2. 

In  the  autonomous  (self-designate)  mode  the  attack  helicopter  is  masked  in  a firing 
position  until  directed  to  engage  enemy  targets . The  attack  helicopter  then  unmasks , 
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identifies  the  target  and  engages  it,  designating  the  target  with  its  on-board 
laser  designator  until  the  missile  impacts.  AAH  may  engage  several  targets  from 
the  same  firing  position  before  moving  to  alternate  firing  positions.  The  attack 
continues  until  directed  otherwise. 

In  the  remote  ground  designation  operational  mode,  targets  for  the  AAH  are  desig- 
nated by  a ground  operator  using  a laser  designator.  After  missile  launch,  the 
attack  helicopter  can  remask,  while  the  designator  operator  continues  to  designate 
the  target  until  missile  impact. 

2.1.3  A Typical  ASH/ AAH  Operational  Procedure  Guide 

Table  2-1  is  a simplified  typical  ASH/ AAH  operational  mode  procedure  guide.  It 
depicts  one  popup  position.  For  multiple  popups  the  procedure  for  the  one  popup 
position  would  be  repeated. 

2.1. A A Typical  AAH  Mission  Profile 

A typical  AAH  mission  profile  is  discussed  in  the  appendix.  The  discussion  includes 
AAH  cruising  speeds  and  altitudes;  distance  traveled  between  various  preselected 
positions;  times  at  the  positions;  hovering  and  firing  altitudes;  firing  ranges; 
the  maximum  AAH  speed  during  search,  tracking,  and  acquisition  modes;  AAH  maneuver- 
ability capabilities;  and  a graphic  portrayal  of  AAH  conducting  a defensive  mission. 

Prior  to  engagement,  aerial  scout  helicopters  provide  coordination  with  ground 
maneuver  units  and  conduct  Initial  reconnaissance  to  select  routes  of  ingress  and 
egress  of  holding  areas,  attack  positions,  and  primary  and  alternate  firing  posi- 
tions for  the  attack  helicopters.  During  the  operation,  the  scout  helicopters 
acquire,  identify,  and  hand  off  targets  and  assist  in  the  movement  of  the  attack 
helicopters.  This  is  done  using  appropriate  terrain  flight  techniques.  The  scout 
helicopters  may  lead  the  attack  helicopters  from  a holding  area  to  the  attack  and 


^Issues  Related  to  Helicopter  Operating  in  Combat  (U) , D.  L.  Hall,  October  1975, 
Northrop  Corporation,  Electro-Mechanical  Division  No.  7292 

2 

AAH  Remote  Aerial  Scout  Designation  Mode  Operational  Analysis  (U) , T.  G.  Baxter, 
July  1976,  Northrop  Corporation,  Electro-Mechanical  Division  No.  7742 
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Table  2-1.  ASH/AAH  OPERATIONAL  MODE 
PROCEDURAL  GUIDE  (ONE  POPUP  POSITION) 


ASH  Observer 

ASH  Pilot 

AAH  Gunner 

AAH  Pilot 

Before 

Arrival 

Check  map 

Check  map 

Check  map 

Check  map 

Check  communications 

Check  communications 

Check  communications 

Check  communications 

Check  sight /laser 

Check  radio  fre- 

Check  sight/seeker 

Check  radio  fre- 

quencles 

1 

quencles 

Arrival  at  Popup  Position 

Sight  on 

Position  aircraft 

- 

Position  aircraft 

Say  when  ready 

- 

Say  when  ready 

- 

Say  to  start 

- 

Say  to  start 

- 

ASH  Popup 

Require  pilot  to 

Unmask 

Hold  position 

Hold  position 

unmask 

(masked) 

(masked) 

Locate  target 

Hold  unmask 

Hold  position 

Hold  position 

(masked) 

(masked) 

Require  pilot  to 

Re mask 

Hold  position 

Hold  position 

remask 

(masked) 

(masked) 

Remasked 

Give  handoff  Infor- 

— 
Hold  mask 

Receive  and  plot 

Hold  mask 

matlon  to  AAH 

target  Information 

gunner 

from  ASH  aerial 

observer 

ASH  Popup 


Require  pilot  to 

Unmask 

Prepare  Information 

Hold 

mask 

unmask 

and  plan  for 
search 

Search  and  acquire 

Hold  unmask  until 

Prepare  information 

Hold 

mask 

target 

countdown 

and  plan  for 
search 

t 


Table  2-1.  ASH/AAH  OPERATIONAL  MODE 
PROCEDURAL  GUIDE  (ONE  POPUP  POSITION)  (Continued) 


ASH  Observer 

ASH  Pilot 

AAH  Gunner 

AAH  Pilot 



ASH/AAH  Popup 

Request  AAH  to  un- 
mask on  command 
"Laser  On" 

Hold  unmask 

Search  for  target 

Unmask  on  command 

Hold  sight  on  target 

Turn  on  laser 

Report  lock-on 

Hold  unmask 

Hold  sight  on  target 

Lase  target 

Report  AAH  detect 

Hold  position 

Hold  sight  on  target 

Lase  target 

Launch  weapon 

Then  remask 

"Verify  smoke"  on 
target  handed  off 

Laser  off 

- 

- 

Remask  after  smoke 
verification 

“ 

“ 

firing  positions.  During  the  attack,  specific  engagement  techniques  from  the 
firing  positions  will  differ,  depending  on  which  of  the  three  operational  modes  is 
being  employed. 

2.1.5  ASH  Aircrew  Functions 

Following  is  an  identification  of  the  ASH  functions  and  the  ASH  aircrew  to  whom 
they  are  assigned: 


Flight 

System 

Mission 

Crewman 

Control 

Communications 

Navigation 

Monitor 

Execution 

Pilot 

X 

X 

Observer 

X 

X 

X 

The  following  conclusions  were  made  in  a recent  government  study: 


Advanced  Scout  Helicopter  Man-Machine  Interface  Investigation  Aviation  Team, 

Human  Engineering  Applications  Directorate,  U.S.  Army  Human  Engineering  Laboratory, 
Aberdeen  Proving  Ground,  Maryland,  September  1975 


2-5 


a.  The  proposed  ASH  provides  the  aircrew  with  excellent  day/night  standoff 
target  acquisition  capability. 

b.  The  analysis  indicates  that  workload  is  anticipated  to  be  high  in  day 
nap  of  earth  (NOE)  operation  and  exceeds  crew  capability  in  night  NOE  operations. 

c.  The  workload  could  be  reduced  by  considering  automation  and  integration 

in  the  flight  control,  navigation,  communication,  and  target  acquisition  functions. 

The  general  recommendation  is:  Specialized  research  is  needed  in  conducting  night 

NOE  operations  using  electronic  displays  to  ascertain  attention  requirements. 

2.1.6  ASH  Mission  Segment  Functional  Block  Diagram 

ASH  mission  functions  consist  of: 

a.  Premission  briefing  and  planning 

b.  Takeoff 

c.  Enroute 

d.  Observation 

e.  Direct  artillery  fire 

f.  Direct  AAH  attack 

g.  Direct  tactical  airstrike 

h.  Return  to  base 
1.  Landing. 

Observation  and  direct  AAH  attack  ASH  mission  segments  are  of  most  interest  for 
the  ASH/ AAH  TADS  vlslonics  application.  An  example  functional  block  diagram  of 
the  observation  portion  of  the  mission  is  shown  in  Figure  2-1. 

2.2  ASH  SURVIVABILITY 

The  appendix  discusses  the  safe  level  of  helicopter  exposure  times  versus  range 
against  various  Soviet  air  defense  systems.  It  also  Indicates  where  these  air  defenses 
are  located  and  what  these  air  defense  systems  rely  upon  for  acquisition  and  detection. 
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Figure  2-1.  OBSERVATION  MISSION  SEGMENT  FLOW  BLOCK  DIAGRAM 

Recommendations  are  made  regarding  the  flying  techniques  employed  against  these 
4 5 

threats.  ’ 

2.3  PARAMETERS  THAT  AFFECT  THE  ASH  TADS  FLIR  IMAGE  QUALITY 

Table  2-2  is  a listing  of  parameters  that  can  affect  the  FLIR  image  quality  of  the 
ASH  TADS.  This  listing  gives  the  parameter,  its  value,  its  security  classification, 
and  the  reference  from  which  it  was  obtained.  The  list  is  intended  to  bound  the 
parameters  being  addressed  in  the  FLIR  Sensor  Processor  Study. 


4 

AAH  Remote  Aerial  Scout  Designation  Mode  Operational  Analysis  (U) , T.  G.  Baxter, 
July  1976,  Northrop  Corporation,  Electro-Mechanical  Division  No.  7742 

^Issues  Related  to  Helicopter  Operating  in  Combat  (U) , D.  L.  Hail,  October  1975, 
Northrop  Corporation,  Electro-Mechanical  Division  No.  7292 
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Table  2-2.  PARAMETERS  THAT  CAN  AFFECT 
ASH  TADS  FLIR  IMAGE  QUALITY 


TADS 'FLIR  Subsystem 
Parameter 

Value 

Classification 

Reference 

FLIR  Spectral  Sensitivity 

8 to  lA  p 

U 

4,5 

Operating  Region 

FLIR  Wide  FOV* 

a>52A  X 698  milliradians  (~30“ 
X -40°) 

U 

4,5 

FLIR  Intermediate  FOV 

175  to  314  milliradians  (10° 
to  18°) 

u 

4,5 

FLIR  Narrow  FOV 

35  to  70  milliradians  (2°  to 
4°) 

u 

4.5 

Nominal  FLIR  Frame  Rate 

30  frames /second 

u 

4,5 

FLIR  Interlace 

2 to  1 

u 

4,5 

Antistreaking  Compensa- 
tion 

FLIR  has  compensation  circuit- 
ry to  eliminate  the  loss  of 
information  encountered  when 
viewing  very  high  contrast 
targets  and  to  prevent  the 
loss  of  terrain  details  when 
viewed  with  a sky  background 

u 

5 

FLIR  Polarity  Reversal 

Operator  selectable  and  no  re- 
adjustment of  contrast  and 
brightness  controls 

u 

4,5 

FLIR  Automatic  and  Semi- 
automatic control 
(l.e.,  brightness. 

Desired  where  feasible  to  re- 
duce operator  task  loading 
and  optimize  image  quality 

u 

4,5 

contrast , and  gain 
controls) 

FLIR  Duration  of  View  - 
Field  Change  Switching 

Field  changes  are  less  than 
1/2  second 

u 

4.5 

and  Settling  Time  for 
View 

FLIR  System  Intensity 

See  Appendix 

c 

5 

Transfer  Curves  for 
Low  Contrast  Setting 

1 

FLIR  SIT  Curves  for  Max- 

See  Appendix 

c 

5 

Imum  Contrast  Setting 

FLIR  Minimum  Resolvable 

See  Appendix 

c 

5 

Temperature  Difference 

FLIR  Optical/Electronic 
Noise 

Not  be  evident  on  the  displays 
and  be  less  than  one  shade 
of  gray 

1 

u 

1 

4.5 

*The  wide  FOV  is  required  to  allow  the  copilot/observer  t 
enroute  checkpoint  navigation. 

:o  assist  the  pilot  with 
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Table  2-2.  PARAMETERS  THAT  CAN  AFFECT 
ASH  TADS  FLIR  IMAGE  QUALITY  (Continued) 


TADS  FLIR  Subsystem 
Parameter 

FLIR  Crosstalk 


FLIR  Dead  and  Noisy 
Channels 


FLIR  Distortion  Across 
Total  FOV 


FLIR  Optical  Focus 


FLIR  Display 


FLIR  Performance 


FLIR  Aspect  Ratio 
(vertical  to  hori- 
zontal) 


Value 

Classification 

Reference 

The  signal  at  the  output  of 
any  nonirradlated  detector 
does  not  exceed  5%  of  the 
signal  in  the  irradiated 
detector  and  no  signal  in 
any  other  channel  that  ex- 
ceeds 5%  of  the  generating 
signal. 

U 

4,5 

Noise  Equivalent  Temperature 
Difference  (NETD)  does  not 
exceed  0.5°C  in  the  center 
20%  of  the  sensor  FOV  and 
throughout  the  rest  of  the 
FOV  the  number  of  inactive 
channels  or  channels  whose 
NETD  exceeds  0.5“C  is  not 
greater  than  3%  of  the 
total  number  of  channels. 
Defective  channels  are 
separated  by  more  than  four 
acceptable  channels. 

U 

4.5 

Does  not  exceed  10%  of  the 
vertical  field  in  either  the 
vertical  or  horizontal  di- 
rection in  the  center  50% 
of  the  FOV  and  20%  of  the 
remainder. 

u 

4,5 

Proper  image  quality  to  be 
maintained  (by  passive  tem- 
perature compensation)  from 
0.5  km  to  infinity  for  all 
FOVs.  Automatic  focus  is 
desired. 

u 

5 

FLIR  video  output  to  drive 
the  displays  required  by 
AMC-SS-AAH-IOOOA. 

u 

5 

The  FLIR  sensor  performance 
parameters  will  meet  the  re- 
quirements cited  in  para- 
graphs 2.1  and  2.2. 

u 

4,5 

3:4 

u 

4.5 

Table  2-2.  PARAMETERS  THAT  CAN  AFFECT 
ASH  TADS  FLIR  IMAGE  QUALITY  (Continued) 


TADS  FLIR  Subsystem 
Parameter 

Value 

Classification 

Reference 

FLIR  Derotation 

The  FLIR  presents  the  gunner 
with  an  erect  image  at  all 
turret  gimbal  angles  with 
respect  to  helicopter  axes. 

U 

5 

TADS  Acquisition  Range 
Parameter 

TADS  Night  Target  Detec- 
tion Ranges 

See  Appendix 

C 

5.6 

TADS  Night  Target  Recog- 
tion  Ranges 

See  Appendix 

C 

5,6 

TADS  Day  Acquisition 
Range 

See  Appendix 

C 

5,6 

TADS  Target  Character- 
istics Parameter 

TADS  Target  Size 

See  Appendix 

C 

5 

Target  Contrast 

See  Appendix 

C 

5 

Night  (target  to 

background) 

See  Appendix 

C 

5 

TADS  Maximum  Target 
Crossing  Velocity 

See  Appendix 

C 

5 

TADS  Atmospheric  Con- 
ditions Parameter 

TADS  Atmospheric  Condi- 
tions 

During  daylight  the  visibility 
is  12  km  and  scene  illumi- 
nance is  1,000  to  100,000 
lumens  per  square  meter. 

For  night  (FLIR)  operation, 
the  atmospheric  transmission 
is  85%  per  kilometer  (8  to 
14  microns) . 

U 

A. 5 

TADS  System  Stabiliza- 
tion and  Tracking 
Parameter 

TADS  System  Stabiliza- 
tion and  Tracking 

Provides  the  resolution  and 
pointing  accuracy  required 
under  the  helicopter's  vi- 
bration and  translational 
and  angular  accelerations. 
Performance  is  not  signifi- 
cantly degraded  under  the 

U 

A. 5 
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Table  2-2.  PARAMETERS  THAT  CAN  AFFECT 
ASH  TADS  FLIR  IMAGE  QUALITY  (Continued) 


TADS  System  Stabiliza- 
tion and  Tracking 
Parameter 


TADS  System  Stabiliza- 
tion and  Tracking 
(Continued) 

Linear  Motion  Compensa- 
tion 

Image  Motion  Compensa- 
tion 

tads  Turret  Slewing 


Classification  Reference 


vibration  and  blast  pressure 
environment  caused  by  weap- 
ons firing. 

Include  in  TADS 


Is  allowable  to  Improve  dy- 
namic resolution 

Slewing  rates  at  least  to 
1.047  radians/second  (60°/ 
second)  In  each  axis  and 
accelerations  of  at  least 
2.094  radians/second 
(120° /second) 

0.5236  radian/second  (30°/ 
second) 


Video  monitor  displays  are 
compatible  with  specifica- 
tions EIA-RS-343  and  EIA- 
RS-330  and  other  require- 
ments for  the  875  and  525 
horizontal  scan  rates , re- 
spectively, and  has  a min- 
imum of  10  shades  of  gray 
(/T  brightness  variation). 


No  vignetting  Is  permitted 
for  the  FLIR  and  TV  subsys- 
tems and  there  is  no  FOV 
cutoff  due  to  obscurations 
within  the  TADS  turret. 


**These  monitor  characteristics  were  omitted  in  the  revised  version  of  this 
document . 


TADS  Tracking 


TADS  Controls  and 
Displays  Parameter 


Monitor  Characteristics 


TADS  Vignetting/ 
Obscuration  Parameter 


Vlgnet  tlng/Obs  curatlon 
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2.4  IMAGE  QUALITY  ANALYSIS 


The  appendix  shows  the  relationship  between  the  number  of  picture  elements  across  the 
target  as  a function  of  range  for  the  three  FOVs;  the  image  motion  in  PELS/second 
versus  range  for  the  three  FOVs;  and  image  motion  and  angle  rate  for  the  three  FOVs. 
It  also  discusses  the  effect  of  temperature  on  the  FLIR  lens  distortion,  and  the 
expected  temperature  ranges  for  the  target-to-background  temperature  differential, 
sky  horizon  temperatures,  and  sea  temperatures. 
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SECTION  3 


FAST  RATIONALIZED  HAAR  TRANSFORM 

This  section  covers  the  basic  definition  of  Haar  functions,  their  extension  to  two- 
dimensional  formats  for  image  processing  applications,  and  their  expression  in  terms 
of  matrices. 

3.1  BASIC  DEFINITIONS 

Haar  transforms  provide  the  basis  for  all  image  processing  techniques  presented  in 
this  report.  Like  other  transforms,  they  can  assign  quantitative  measures  to  quali- 
tative aspects  of  the  original  image,  or  replace  an  involved  measurement  technique 
by  an  easier  and  faster  one.  These  measures  in  turn  identify  the  Important  vari- 
ables for  processing  image  restoration  and  enhancement. 

Haar  transforms  are  also  comparatively  simple.  Each  transform  component  effectively 
multiplies  the  original  function  by  either  0,  +1,  or  -1.  Furthermore,  they  combine 
the  abilities  of  local  transforms  (e.g. , Block  transforms)  to  touch  up  given  areas 
of  a picture,  with  those  of  global  transforms  (e.g.,  Walsh  transforms)  to  render 
broad  comprehensive  interpretation  of  data. 

The  general  Haar  function  h(r,m,x)  appears  in  Figure  3-1. 

Sequency  r bears  a certain  resemblance  to  frequency.  Offset  m ranges  between  1 and 
2^,  These  mutually  orthogonal  functions  can  be  rationalized  by  reducing  all  ampli- 
tudes 2^^^  to  a common  value  of  unity.  A weighting  function  of  2*^,  however,  must 
then  be  Included  in  either  the  forward  or  Inverse  transform. 


_L-r 

-1/2 

1 

( 

1 

h’' 

T - 

^ f 
1 

1- m.2'' 

- 

4170 

Figure  3-1.  GENERAL  HAAR  FUNCTION  h(r,m,x) 
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3.2  TWO-DIMENSIONAL  FORMATS 


Haar  functions  can  be  used  to  expand  general  one-dimensional  waveforms.  Two  dimen- 
sional waveforms,  such  as  image  intensity  distributions  I(x,y)  can  similarly  be 
expanded  in  a product  series: 


I(x,y) 


2 


a 

rmpn 


r,m,p,n 


h(r,m,x)h(p,n,y) 


a 

rmpn 


1/2  1/2 

/ / 


-1/2  -1/2 


I(x,y)h(r ,m,x)h(p,n,y)dxdy 


(1) 


(2) 


While  this  applies  to  primary  domain  -1/2  ^ x ^ 1/2,  -1/2  ^ y ^ 1/2,  simple 
linear  scale  changes  will  qualify  these  relations  for  images  of  any  finite  extent. 

A rationalized  two-dimensional  product  h(r,n,x)h(p,n,y)  appears  as  in  Figure  3-2. 


1 ■ ~ ffv2'^-  * 

1 

n.2-P 

0 

0 

T 

0 

♦1 

-1 

0 

1 

0 

-1 

♦ 1 

0 

0 

111 

0 
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Figure  3-2.  GENERAL  TWO-DIMENSIONAL  RATIONALIZED  HAAR  PRODUCT  FUNCTION 

When  all  such  diagrams  have  been  located  in  rectangular  (r,p)  coordinate  space  as 
in  Figure  3-3,  the  resulting  array  represents  the  Haar  image  transform  domain. 
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Figure  3-3.  HMR  IMAGE  DOMAIN 
A detailed  picture  of  the  array  is  given  in  Figure  3-4. 
3.3  MATRIX  REPRESENTATION 


Forward  and  inverse  Haar  transforms  in  equations  (1)  and  (2)  can  also  be  expressed 
in  terms  of  matrices.  It  should  be  noted  that  matrix  expressions  substitute  dis- 
crete summation  for  integration  in  equation  (2).  With  this  proviso,  continuous 
image  distribution  I(x,y)  reduces  to  the  discrete  expression  I(tAx,  lAy)  where  Ax 
and  iy  typically  assume  values  smaller  than  the  resolution  spot  size  of  the  imaging 
system.  Integers  t and  1 span  a range  sufficient  to  cover  maximum  and  minimum 
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rKXXv) 
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TWO-DIMENSIONAL  RATIONALIZED  HAAR  FUNCTIONS 


values  of  x and  y.  This  expression  In  turn  can  be  represented  by  the  matrix  I 
or  simply  I. 


Haar  function  h(r,m,x)  reduces  to  the  matrix  h^^  where  integer  s designates  succes- 
sive Haar  functions  in  a particular  sequency  and  then  moves  on  to  the  next  sequency. 
Using  only  the  first  few  sequencies  in  a rationalized  format  to  clarify  the  illus- 
tration: 


rationalized 


1 

1 

1 

0 

-1 

0 

0 

0 


1 1 

1 1 

-1  -1 

0 0 

0 0 

1 -1 

0 0 

0 0 


1 

-1 

0 

1 

0 

0 

1 

0 


1 1 
-1  -1 


0 

1 

0 

0 

-1 

0 


0 

-1 

0 

0 

0 

1 


1 

-1 

0 

-1 

0 

0 

0 

-1 


fh,  ] J (3) 

ts  rationalized 


Expansion  coefficient  a likewise  reduces  to  matrix  [a  1 

rmpn  ‘ sq 

Equations  (1)  and  (2)  then  convert  to  the  matrix  expressions 


HAH 


H IH 


(4) 

(5) 


Because  of  the  simplicity  of  (3)  and  the  compactness  of  (4)  and  (5) , the  matrix 
approach  will  be  preferred  in  general. 
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SECTION  4 
IMAGE  DESCRIPTORS 


4..1.  OVERVIEW 

Although  mean  square  error  traditionally  has  been  used  as  a fidelity  criterion  for 
most  image  coding  techniques,  there  are  some  areas  in  which  it  does  not  compare  well 
with  human  evaluation.  As  an  example,  mean  square  error  does  not  make  sense  in  the 
areas  of  image  enhancement  or  blind  restoration,  and  in  most  cases  it  cannot  be 
obtained  because  no  reference  image  is  available.  A quantification  of  the  effect  of 
various  image  enhancement  and  restoration  techniques  is  required  that  agrees  well 
with  the  human  subjective  evaluation  and  can  be  obtained  easily  from  the  image. 

In  order  to  facilitate  the  evaluation  of  images  and  image  processing  algorithms 
throughout  the  course  of  this  study,  a set  of  quantitative  image  descriptors  has 
been  developed.  The  descriptors  basically  summarize  the  information  of  the  image  by 
dividing  it  into  two  major  categories,  tone  and  activity.  The  tone  of  an  image  is  gov- 
erned by  the  diversity  of  the  local  ensemble  averages  of  the  gray  levels  of  the  image 
and  is  important  in  distinguishing  objects  within  the  scene.  Activity  is  determined  by 
the  way  the  gray  levels  change  from  pel  to  pel  and  is  a measure  of  how  busy  the  scene 
is.  In  the  Haar  domain,  these  two  features  may  be  measured  directly.  Tone  is 
equivalent  to  the  variance  of  the  dc  terms,  a ^ ^ ^ , of  each  16  x 16  pel  block  and 

activity  is  the  total  ac  energy  in  the  transform  domain. 

The  division  of  image  information  into  only  two  categories,  tone  and  activity,  is  not 
sufficient  for  useful  analysis,  however.  Two  images  may,  for  example,  have  compa- 
rable activity  when  one  contains  significant  edges  (outline  of  a tank,  etc.)  while 
the  other  contains  only  high-power  white  noise.  For  this  reason,  image  activity  is 
further  subdivided  into  texture  activity  and  edge  activity. 

Texture  in  the  image  is  characterized  by  a series  of  repetitious  gray  level  transi- 
tions (edges)  that  are  usually,  though  not  necessarily,  of  relatively  low  power. 

Edge  Information,  on  the  other  hand,  is  typified  by  a smaller  number  of  gray  level 
transitions,  usually  of  higher  power.  These  two  characteristics  are  exhibited  in  the 
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individual  sequency  groupings  of  the  Haar  domain  as  broadly  distributed  power  in  the 
case  of  texture  activity  and  as  power  spikes  or  singularities  in  the  case  of  edge 
activity.  By  applying  an  edge  criterion  function  to  each  transform  coefficient 
within  a sequency  group,  the  activity  (ac  power)  due  to  edge  may  thus  be  separated 
from  the  nonedge  activity,  which  by  default  is  due  to  texture. 

4.2  USE  OF  DESCRIPTORS 

The  descriptors  will  be  used  to  quantify  the  characteristics  of  images  both  before 
and  after  the  application  of  various  image  processing  algorithms.  The  resultant 
changes  in  the  image  descriptors  can  then  be  used  to  detect  the  net  effect  of  proc- 
essing on  the  image. 

Changes  in  the  descriptors  can  be  presented  in  a more  general  format  if  the  three 
individual  descriptors  are  viewed  as  the  components  of  a three-element  descriptor 
vector.  As  an  example,  let  T,  E,  and  X be  the  values  of  an  image's  tone,  edge 
activity,  and  texture  activity  descriptors,  respectively,  before  processing.  The 
image's  descriptor  vector,  ID,  would  be  given  by  the  column  vector  ^ = [T,  E,  X]. 

If  T' , E',  and  X'  are  the  corresponding  descriptor  values  and  is  the  descriptor 
vector  after  processing,  the  change  in  the  image  can  be  typified  by  the  difference 
vector  D “ ^ - I^' . 

Utilizing  the  difference  vector  D allows  examination  of  the  net  amount  of  change 
given  by  |d|  , as  well  as  the  direction  of  change.  Direction  of  change  would  be 
determined  by  the  slopes  of  various  projections  of  D onto  planes  of  interest  in 
image  descriptor  space.  Measuring  the  direction  of  change  is  expected  to  be  very 
useful  as  it  should  lend  to  the  development  of  algorithms  that,  for  example,  affect 
edge  more  than  texture. 

4.3  DESCRIPTORS 

Since  the  primary  research  tool  for  this  sensor  processor  study  performs  the  Haar 
transform  on  individual,  nonoverlapping,  16  x 16  pel  blocks,  local  descriptors  must 
first  be  obtained  for  each  block,  and  then  combined  to  form  global  descriptors  that 
apply  to  the  entire  image  or  areas  of  Interest  within  the  image.  The  definitions  of 
the  local  descriptors  and  their  rules  of  composition  to  form  the  global,  or  image, 
descriptors  are  presented  here. 
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4.3.1  Local  Descriptors 


4. 3. 1.1  Local  Tone  - The  local  tone  of  a block  is  the  dc  term  of  its  Haar  trans- 
form. In  the  sensor  processor's  version  of  the  rationalized  Haar  transform,  the 
value  of  this  term  is  equivalent  to  the  average  gray  level  of  the  image  block.  Let 

t.  be  the  local  tone  of  the  1^^  block,  t.  » a , r = 1 , m » 1 , p “ 1 , n = 1 . The 
i i rmpn 

term  a^  is  in  general  the  rmpn^'^  term  of  the  block.  In  this  case  a^  is  the 
rmpn  rmpn 

dc  term. 


4. 3. 1.2  Local  Activity  - The  local  activity  of  the  i^^  block  is  its  ac  energy. 
This  is  given  by  the  sum  of  the  squares  of  the  ac  terms  of  its  Haar  transform.  Let 
c^  denote  the  local  activity  of  block  i, 


c 


i 


A 


nil 

rmpn 

rmpn 


2 


The  term  c^  by  itself  is  insufficient  to  characterize  the  information  in  block  i. 

For  this  reason  ac  energy  is  divided  into  edge  activity  and  texture  activity. 

4. 3. 1.3  Local  Edge  Activity  - The  local  edge  activity  of  the  1^^  block  is  the  ac 
energy  that  satisfies  the  edge  criterion  function.  In  subjective  terms,  the  edge 
criterion  function  says  that  if  the  power  of  an  ac  coefficient  contributes  more  than 
its  fair  share  to  the  total  power  of  its  sequency  group,  then  that  term  is  the  result 
of  an  edge  in  the  image  domain.  More  specifically,  if  f denotes  the  set  of  edge 
terras  then 


if 


a^  e f 
rmpn 


^rmpn 

II 

rmpn 

mn 
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sequency  group  and  is  the 


th 

where  a is  the  edge  power  threshold  for  the  rp 

i i 

relative  entropy  of  the  group.  If  P(a  ) is  the  probability  that  a will  occur 

rmpn  rmpn 

within  the  group,  then 


P(a^  ) log  P(a^  ) 
mn  rmpn  2 rmpn 


rp 


log^  (number  of  terms  in  rp 


th 


group) 


The  values  of  a are  shown  in  Table  4-1.  As  can  be  seen  in  the  table,  a = a . 

rp  rp  pr 

This  reflects  the  assumption  that  horizontal  and  vertical  edges  are  equally  likely. 

The  values  of  were  chosen  empirically,  but  approximate  1/1+K  where  K is  the 
nominal  number  of  Haar  functions  in  the  rp  group  to  which  a straight  edge  would  con- 
tribute power.  This  approximate  relationship  for  allows  diagonal  edges  to  be 
detected  more  readily. 


The  purpose  of  weighting  a with  the  group's  relative  entropy  can  be  seen  by  exami- 

r P . 

ning  the  two  representative  histograms  of  coefficients  of  the  rp  group  shown  in 
Figure  4-1. 


The  power  that  the  edge  terms  would  require  in  order  to  differ  significantly  from 
the  other  terms  is  greater  in  the  flat  distribution  of  Figure  4-lB  than  in  the  dis- 
tribution of  Figure  4-lA.  A good  measure  of  the  distribution  flatness  is  its  entropy. 
Its  entropy  is  therefore  used  to  weight  the  edge  power  threshold. 


If  we  let  e^  be  the  local  edge  activity,  then 


n - nn 

rmpn 


(a  ) 

rmpn 


2 


if 


a^  E T 
rmpn 


The  terms  a_^  1-11  excluded  from  the  above  expression  by  assuring  that  they 

will  never  satisfy  the  edge  criterion  function. 
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B - FLAT  DISTRIBUTION 


4. 3.1.4  Local  Texture  Activity  - In  this  study,  local  texture  activity  is,  by 
definition,  the  ac  energy  remaining  in  the  block's  Haar  transform  after  all  the  edge 
terms  have  been  removed.  If  is  the  local  texture  activity  for  the  i*"^  block,  then 


i ^ nzi 


if 


rmpn 


rmpn 


4. 3.1. 5 Examples  of  Local  Descriptors  - Two  sample  images  shown  in  Figure  4-2, 
"U.S.C.  Girl"  and  "Moon,"  were  selected  for  initial  evaluation  of  the  above  descrip- 
tors. It  was  desirable  for  the  test  data  set  to  be  kept  as  small  as  possible  for 
development  purposes  and  yet  be  sufficiently  diverse  to  exhibit  a reasonable  cross 
section  of  image  characteristics.  It  is  felt  that  the  two  selected  Images,  Girl  and 
Moon,  provide  enough  local  diversity  individually  and  global  diversity  between  each 
other  to  form  a good  initial  test  data  set. 


Each  image  consists  of  256  x 256  pels  that  are  subdivided  into  a 16  x 16  array  of 
16  X 16  pel  blocks.  The  local  descriptors  were  computed  for  each  block  of  both 
sample  images.  The  resultant  arrays  of  local  descriptors  are  given  in  Tables  4-2 
through  4-5. 


Examination  of  the  arrays  pertaining  to  the  local  tone  of  the  Girl  and  Moon  Images 
(Table  4-2)  shows  that  the  results  obtained  are  in  good  agreement  with  subjective 
evaluation.  This  is  to  be  expected  since  the  local  tone  is  nothing  more  than  the 
average  gray  level  of  the  16  x 16  pel  block.  Dark  areas  of  each  picture  have  a 
correspondingly  lower  local  tone  than  bright  areas.  Subjective  agreement  is  also 
good  for  local  activity  (Table  4-3).  Busier  areas  of  each  picture  have  correspond- 
ingly higher  values  of  activity. 

Two  blocks  that  both  exhibit  high  activity  and  yet  contain  significantly  different 
information  are  flagged  by  asterisks  in  Table  4-3A.  The  single  asterisk  value 
relates  to  a block  containing  the  edge  of  the  girl's  face  while  the  double  asterisk 
value  pertains  to  a block  containing  the  flower.  These  two  blocks  provide  good 
examples  of  edge  and  texture.  The  local  texture  activity  values  of  the  same  two 
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Figure  4-2.  SAMPLE  IMAGES 
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Table  4-5A.  LOCAL  EDGE  ACTIVITY  ACROSS  GIRL  IMAGE 
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blocks  are  similarly  flagged  In  Table  4-4,  Note  that  the  value  of  texture  activity 
is  much  greater  for  the  flower  sector  than  for  the  edge  sector.  Table  4-5  shows 
that  the  edge  activity  is  likewise  isolated  in  the  edge  sector. 

4.3.2  Global  Image  Descriptors 

Obtaining  global  descriptors  of  tone,  edge  activity,  and  texture  activity  requires 
a suitable  combination  of  the  quantities  computed  on  the  subpictures  of  the  image. 
Since  the  edge  and  texture  activity  add  up  to  local  activity  numerically,  only  the 
results  of  local  tone  and  activity  will  be  developed  herein.  The  rules  for  the 
combination  of  local  edge  and  texture  activity  will  be  equivalent  to  those  of  local 
activity. 

The  local  tone  and  activity  have  been  computed  on  all  of  the  256  blocks  of  sample 
Images  girl  and  moon.  The  arrays  of  the  local  values  of  tone  and  activity  were  given 
in  Tables  4-2  and  4-3.  A single  number  must  be  computed  from  each  of  the  arrays  that 
characterizes  the  basic  differences  between  the  two  images.  To  aid  in  the  comparison 
of  the  images,  some  first  order  statistics  of  the  local  arrays  were  computed  as 
shown  in  Table  4-6. 


Table  4-6.  STATISTICS  OF  LOCAL  TONE  AND  ACTIVITY 


Girl 

Moon 

Local  Tone 

Mean 

73 

127 

Minimum 

10 

30 

Max imum 

205 

178 

Standard  deviation 

35 

22 

Local  Activity 

Mean 

2,367 

2,065 

Minimum 

130 

489 

Maximum 

23,606 

9,975 

Standard  deviation 


3,189 

1,391 
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4. 3.2.1  Image  Tone  - As  can  be  seen  from  Table  4-6,  the  mean  of  the  local 
tone  (average  brightness)  is  greater  for  moon  than  for  girl.  Subjective  evalua- 
tion, however,  indicates  that  the  girl  image  has  the  most  tone.  In  agreement  with 
the  concept  that  tone  and  brightness  should  be  different  quantities  on  any  given 
picture,  the  standard  deviation  of  the  local  tone  is  taken  as  a measure  of  the  global, 
or  image,  tone.  Let  the  image  tone  be  denoted  by  T,  then 


T 


A 


where  t^  is  the  local  tone  of  the  i^^  block,  B is  the  average  brightness  of  the 
image,  and  N is  the  number  of  16  x 16  pel  blocks  in  the  image.  The  source  of  the 
difference  in  standard  deviations  of  the  local  tone  of  the  two  Images  can  be  seen 
readily  by  examining  the  histograms  of  local  tone  shown  in  Figures  4-3  and  4-4. 


18.9  46.3  73.6  100.9  128.3  15S.6 

AMPLITUDE,  A 4170 

Figure  4-3.  HISTOGRAM  OF  LOCAL  TONE  OF  GIRL 
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30.0  59.5  89.1  118.7  148.2  177.8 

AMPLITUDE.  A 4170 

Figure  4-4.  HISTOGRAM  OF  LOCAL  TONE  OF  MOON 


The  preceding  expression  for  image  tone  exhibits  the  following  characteristics: 

When  the  local  tone  of  each  image  block  is  approximately  equal,  l.e.,  the  total 
picture  is  of  uniform  average  intensity,  the  image  tone  is  near  zero.  When  the 
local  cone  changes  significantly  from  block  to  block,  the  image  tone  is  large. 

4. 3.2.2  Image  Activity  - A comparison  of  the  statistics  of  the  local  activity  of 
the  girl  and  moon  Images  (Table  4-6)  reveals  a significant  difference  in  their 
standard  deviations.  The  standard  deviation  of  the  local  activity  is  not,  however, 
taken  as  a measure  of  image  activity.  The  reason  for  this  decision  becomes  clearer 
when  we  consider  the  following  example:  In  a given  image  assume  that  each  16  x 16 

pel  block  contains  either  sufficient  edge  or  texture  information  to  give  it  a local 
activity  of  a^,  a constant.  Each  block  may  be  completely  different  from  its  neigh- 
bors; the  only  similarity  is  their  local  activity.  If  the  standard  deviation  of  the 
local  activity  were  the  measure  for  image  activity,  the  image  activity  would  be  zero. 
But,  it  one  half  of  the  blocks  of  the  above  image  were  changed  to  a uniform  gray, 
l.e.,  made  to  contain  no  a-c  energy,  the  standard  deviation  of  the  local  activity 
would  increase.  Such  a result  is  counterintuitive  to  a measure  of  image  activity. 


4-15 


The  measure  to  be  used  for  Image  activity  is  the  mean  of  the  local  activity.  Image 
activity  thus  measured  would  rise  and  fall  in  agreement  with  subjective  evaluation 
in  the  example  given. 

The  standard  deviation  of  the  local  activity  does,  however,  Impart  some  information 
about  the  total  image.  This  parameter  is  dubbed  "image  diversity"  but,  along  with 
image  brightness,  is  not  presently  considered  a necessary  element  of  the  global 
image  descriptors.  Further  evaluation  of  the  descriptors  on  a larger  set  of  Images 
may  alter  this  conclusion. 


Examination  of  Table  4-6  shows  that  the  girl  image  is  about  15  percent  more  active 
than  the  moon  image.  Let  A denote  the  global,  or  image,  activity. 


A 


A 


a 


1 


where  a^  is  the  activity  of  the  i^^  block.  Similar  expressions  apply  for  X,  the 
image  texture  activity  and  E,  the  image  edge  activity;  i.e. , 


X 


E 


A 


A 


where  x^  and  e^  are  the  local  texture  and  edge  activity  respectively. 

The  statistics  of  the  division  of  local  activity  into  edge  and  texture  activity  for 
the  two  sample  images  are  shown  in  Table  4-7. 

lable  4-7  shows  that  the  global  texture  activities  of  the  two  Images  are  comparable 
but  the  girl  image  displays  more  edge  activity.  This  is  in  good  agreement  with 
Initial  subjective  analysis. 
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Table  4-7.  STATISTICS  OF  LOCAL  EDGE  AND  TEXTURE  ACTIVITY 


Girl 

Moon 

Local  Edge  Activity 

Mean 

1,191 

850 

Maximum 

15,147 

4,937 

Minimum 

99 

195 

Standard  deviation 

1,754 

737 

Local  Texture  Activity 

Mean 

1,176 

1,214 

Maximum 

14,675 

5,783 

M Lnimum 

5 

223 

Standard  deviation 

1,635 

776 

4. 3, 2. 3 Global  Descriptor  Vector  - The  three  global  features  used  as  image  descrip- 
tors are  T,  image  tone;  E,  image  edge  activity;  and  X,  image  texture  activity.  These 
three  descriptors  are  defined  to  be  the  components  of  the  image  descriptor  vector, 

ID.  ID  • [T,E,X].  Thus,  the  descriptors  of  an  image  define  a point  in  a three- 
dimensional  image  descriptor  space.  The  concept  of  the  image  descriptors  describing 
points  in  space  will  prove  useful  in  describing  the  distances  and  directions  between 
the  image  descriptor  vectors  of  original  and  processed  images. 

4.4  SUMMARY  AND  CONCLUSIONS 

A vector  of  image  descriptors  has  been  developed.  The  components  of  the  vector 
correspond  to  the  image  tone,  edge  activity  and  texture  activity.  The  sum  of  edge 
and  texture  activity  results  in  the  image  activity  in  general.  The  image  activity 
can  be  obtained  directly  from  the  image  by  ensemble  averaging  the  local  a-c  energy 
of  the  image  blocks  of  interest.  The  separation  of  local  activity  into  edge  and 
texture  activity  is  accomplished  by  studying  the  statistics  of  groups  of  coefficients 
in  the  two-dimensional  Haar  domain. 

The  image  tone  is  obtained  by  computing  the  standard  deviation  of  the  local  tone 
(brightness)  of  the  image  blocks  of  interest. 
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The  descriptors,  as  defined,  provide  valuable  measures  of  the  qualitative  differences 
and  similarities  of  the  two  test  Images.  However,  as  the  program  progresses  and  an 
increasing  set  of  FLIR  images  are  evaluated,  some  of  the  aspects  of  the  descriptor 
set  may  have  to  be  changed.  For  example,  at  the  local  level,  the  set  of  group  edge 
weights,  used  in  the  edge  criterion  function  may  have  to  be  optimized  to  better 

suit  FLIR  image  characteristics.  At  the  global  or  image  level,  features  such  as 
image  diversity  or  brightness  may  prove  to  contain  vital  information  and  therefore 
be  included  as  components  of  the  descriptor  vector. 

The  use  of  general  vector  concepts  in  the  comparison  of  the  Image  descriptors  of 
different  Images  is  expected  to  provide  insight  into  not  only  the  magnitude  of 
change  induced  by  the  application  of  an  image  processing  algorithm,  but  to  the 
direction,  or  quality,  of  change  as  well. 
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SECTION  5 
EXPERIMENTAL  PLAN 


This  section  describes  the  basic  plan  or  approach  for  conducting  the  various  experi- 
ments and  computer  simulations  required  to  gather  test  data.  It  is  hoped  that  this 
data  will  provide  an  insight,  and  ultimately  a solution,  to  the  problems  associated 
with  present  day  Imaging  FLIR  systems. 

5.1  GENERAL  APPROACH 

All  experiments  will  be  conducted  utilizing  either  one  of  two  system  configurations: 
(1)  Northrop 's  two-dimensional  sensor  processor  in  conjunction  with  an  instrumenta- 
tion system,  or  (2)  computer  simulation  on  Northrop 's  Data  Processing  Center  IBM  370 
computer. 

The  experimental  setup  utilizing  the  two-dimensional  sensor  processor  is  preferred 
because  (1)  it  processes  imagery  in  near  real  time,  and  (2)  it  has  a CRT  display. 
These  two  capabilities  provide  the  system  with  features  that  model  more  closely  the 
Advanced  Scout  Helicopter  (ASH)  vislonics.  For  example,  image  motion  and  psycho- 
visual effects  of  Images  displayed  on  a CRT  monitor  (scan  format,  scan  rate,  etc.) 
would  be  encountered  in  the  operation  of  the  FLIR  system  installed  on  ASH. 

5.1.1  Two-Dimensional  Sensor  Processor 

The  majority  of  the  experiments  will  be  conducted  on  the  experimental  setup  depicted 
in  Figure  5-1.  The  setup  Includes  Northrop 's  two-dimensional  sensor  processor  shown 
in  Figure  5-2,  an  instrumentation  subsystem,  and  assorted  ancillary  equipment. 

The  two-dimensional  sensor  processor  has  the  capability  of  accepting  both  static 
Imagery  in  the  form  of  photographic  transparencies,  or  dynamic  imagery  recorded  on 
a one-inch  video  tape.  (Recorded  video  must  be  in  RS170,  525-line,  30-f rame-per- 
second  format  and  be  compatible  with  the  IVC  800A  tape  recorder/reproducer . ) 
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Figure  5-1.  EXPERIMENTAL  TEST  CONFIGURATION 
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Figure  5-2.  TWO-DIMENSIONAL  SENSOR  PROCESSOR  FUNCTION  FLOW  DIAGRAM 


The  transform  section  of  the  two-dimensional  sensor  processor  accepts  the  video 
signals  from  either  the  video  camera  or  the  video  tape  unit,  and  digitizes  it 
linearly  to  an  8-bit  resolution  (256  shades  of  gray).  The  digitized  video  is  then 
converted  into  the  Haar  domain  (in  two  dimensions)  by  using  a fast  rationalized  Haar 
transform.  The  resultant  transform  coefficients  can  now  be  altered  by  a limited 
number  of  hardware  implemented  algorithms  by  using  the  processor's  front  panel 
controls. 

The  transformed  Imagery,  whether  modified  or  not,  is  accepted  by  the  instrumentation 
subsystem.  It  includes  a refresh  memory,  a minicomputer,  a digital  magnetic  tape 
transport,  and  a medium  speed  printer.  *The  instrumentation  system  provides  the 
capability  to  record,  refresh,  and  input  the  transform  coefficients  of  the  image. 
Recorded  coefficients  will  be  analyzed  off-line  to  describe  the  statistical  contents 
of  an  image.  The  resultant  descriptors  can  then  be  used  for  analysis  of  Imagery 
data  and  for  evaluating  the  effects  of  applied  algorithms.  They  can  also  serve  as  a 
basis  for  assessing  the  progress  and  achievements  reached  in  development  programs. 

The  refresh  memory  between  the  sensor  processor  and  the  minicomputer  provides  the 
capability  to  snatch  a limited  area  of  the  transformed  image  (approximately  3 percent 
of  the  total  image).  This  data  may  be  input  to  the  minicomputer  for  modification 
beyond  the  present  capacity  of  the  processor  front  panel  controls.  Such  modification 
is  performed  by  the  minicomputer's  software  and  is  not  in  real  time.  The  portion  of 
the  image  modified  by  the  minicomputer  is  then  stored  and  recirculated  in  the  refresh 
memory  to  refresh  the  TV  monitors. 

The  inverse  section  of  the  two-dimensional  sensor  processor  converts  the  Haar  trans- 
form of  the  image  back  into  a video  signal  that  is  displayed  on  the  TV  monitors. 

Two  TV  monitors  are  provided  to  enable  a person  to  view  both  the  original  image  and 
the  modified  image  simultaneously  for  comparison. 

Either  the  original  or  modified  image  may  be  displayed  on  a small  5-inch  CRT  monitor 
with  a photograph  camera  attached.  This  allows  for  obtaining  snapshots  of  test 
images . 
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5.1.2  Computer  Simulations 


Some  computer  simulations  will  be  required  where  the  hardware  limitations  of  the 
two-dimensional  sensor  processor  preclude  the  development  of  some  concepts  otherwise 
preferred.  The  IBM  370  computer  will  also  be  utilized  to  solve  more  complex  mathe- 
matical problems  such  as  obtaining  a Haar  expression  or  approximation  of  a blur  that 
is  expressed  in  the  Fourier  domain. 

The  computer  simulation  setup  depicted  in  Figure  5-3  includes  an  IBM  370,  centrally 
located  at  Northrop 's  Electronics  Data  Processing  Center,  with  remote  access  termi- 
nals (RAX)  located  at  each  corporate  division. 

Simulation  programs  and  digitized  Imagery  are  input  to  the  simulation  setup  at  the 
370  computer  via  a magnetic  tape  unit.  The  various  algorithms  are  entered  at  the 
RAX  terminals.  Data  outputs  resulting  from  application  of  algorithms  to  imagery 
data,  including  status  and  check  comments,  are  directed  to  the  RAX  terminal  printer 
for  evaluation.  Data  outputs  are  also  directed  to  the  IBM  magnetic  tape  unit  for 
subsequent  off-line  composition  into  photograph  form.  Intermediate  results  of  the 
simulations  may  also  be  input  to  the  sensor  processor  via  the  sensor  processor 
Instrumentation  subsystem's  digital  tape  transport  (Figure  5-2).  This  allows  "quick 
look"  examinations  of  processed  imagery  on  a CRT  display. 


1 

• PROGRAM  ENTRY 

• BATCH  DATA  ENTRY 

• DIGITIZED  IMAGERY  41  70 


Figure  5-3.  COMPUTER  SIMULATION  SETUP 
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5.2  EXPERIMENTAL  TASKS/ STUDIES  OVERVIEW 


The  objectives,  conceptual  solutions  and  generalized  experimental  approach  to  prin- 
cipal areas  in  which  Haar  transform  processing  can  improve  FLIR  Imaging  are  detailed 
in  the  following  paragraphs.  Many  of  the  original  concepts  were  developed  under 
Northrop' s CY  75  and  CY  76  Independent  Research  and  Development  programs.  Specific 
areas  covered  Include  autofocus,  autodeblur,  image  enhancement,  adaptive  signal 
processing  and  data  compression. 

5.2,1  Autofocus 


5. 2. 1.1  Objective  - Because  severe  demands  are  placed  on  a two-man  helicopter 
team,  the  need  for  manual  FLIR  image  refocusing  can  pose  a formidable  problem.  FLIR 
defocusing  arises  from  continual  changes  in  observation  range  as  well  as  from  other 
Incidental  causes.  Automatic  refocusing  offers  a means  to  combat  these  tendencies 
and  relieve  part  of  the  manpower  burden.  The  apparatus  should  sense  the  defocus 
condition  and  produce  a corresponding  error  signal  for  automatic  correction.  This 
signal  might  be  used  only  on  a brief  periodic  basis,  to  minimize  mechanical  wear. 

5. 2. 1.2  (See  addendum  to  this  report) 

5. 2. 1.3  Experimental  Approach  - Experimental  procedure  generally  consists  in 
varying  the  focus  of  the  input  TV  camera  and  noting  the  effect  at  the  output  of  the 
Haar  transform  processor.  The  camera  may  be  directed  at  a test  pattern  which  em- 
phasizes a particular  resolution  spot  size.  This  pattern  should  extend  only  over  a 
single  16-by-16  pel  subsection  of  the  image  plane. 

Under  preliminary  In-focus  conditions,  the  boundary  limits  derived  from  the  proc- 
essor output  should  be  checked  against  the  pattern  spot  size.  For  completeness,  all 
Haar  coefficient  amplitudes  should  be  measured. 

The  camera  can  then  be  defocused  by  varying  amounts  to  either  side  of  the  ideal 
setting  and  the  coefficients  again  measured.  Graphs  of  variations  of  meaningful  Haar 
coefficients  with  different  defocus  settings  can  then  be  constructed.  These  should 
have  the  general  shape  of  Figure  5-4  (see  addendum).  They  will  effectively  convert 
the  dither  sensitivity  of  the  system  into  an  equivalent  (non-linear)  transfer  func- 
tion about  which  a corrective  feedback  loop  may  be  djesigned. 
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Real  photographic  scenes  can  then  be  substituted  for  the  test  pattern,  and  the  above 
process  repeated.  Here,  however,  the  object  will  be  to  determine  whether  different 
scenes  generate  the  same  type  graph  as  that  illustrated  in  Figure  5-4  (see  addendum), 
so  that  a given  corrective  design  might  serve  a large  variety  of  possible  situations. 
To  this  end,  five  photographs  showing  strong  mutual  diversity  might  be  selected. 

5.2.2  Autodeblur 

5. 2. 2.1  Objective  - Pertinent  problem  areas  Include  atmospheric  blurring,  depth 
of  field  and  motion  blur.  Each  one  can  seriously  degrade  image  resolution. 

Atmospheric  blurring  can  arise  from  haze,  thermal  gradients,  clear  air  turbulence, 
and  related  causes.  These  produce  dynamically  changing  random  refractive  index 
gradients  that  degrade  the  spatial  coherence  of  wavefronts  radiated  by  the  object. 

Depth-of-f leld  blurring  concerns  the  spread  of  slant  ranges  to  different  parts  of  a 
large  obliquely  viewed  target,  and  the  inability  to  bring  them  all  into  focus  simul- 
taneously. Low  altitude  reconnaissance  typifies  this  situation. 

Motion  blur  refers  both  to  jittered  and  smeared  images.  Even  if  perfect  correction 
were  performed  for  smeared  images,  the  extremely  rapid  motion  associated  with  smeared 
images  does  not  permit  the  eye-brain  combination  to  view  them  comprehensively  in 
real  time.  Because  of  this  limitation,  smearing  will  not  be  considered  at  this  time. 
Jittering  due  to  underdamped  gyros  or  servo  stability  problems,  however,  may  benefit 
from  Haar  transform  concepts. 

5. 2. 2. 2 Atmospheric  Blur  - 

5. 2. 2. 2.1  Concept  - Atmospheric  blurring  effectively  convolves  unknown  blur  func- 
tion b(x,y)  with  ideal  image  i(x,y).  Representative  shapes  of  b(x,y)  include  a 
pillbox,  a cone  or  any  hill-like  form  which  extends  over  several  pels.  This  extent, 
or  footprint,  is  typically  small  compared  to  the  overall  image  dimensions.  Con- 
volution uses  this  footprint  like  a broad  brush,  to  trace  out  pictorial  details  in 
i(x,y).  The  coarsened  result  is  the  blurred  picture.  As  footprint  size  Increases, 
therefore,  the  blurring  becomes  progressively  worse. 

Autodeblurring  consists  in  identifying  b(x,y),  and  using  the  information  to  cancel 
its  effect.  Identifying  b(x,y)  however  requires  some  procedure  for  separating  it 
from  l(x,y),  since  the  two  functions  always  appear  together.  This  procedure  derives 
from  the  expression 
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b(x,y) 


= (const. )f"^  [J,  . J,  ...  J ... 

L z n N 


-1 


Inverse  Fourier  transform 


J (Pi<l)  = Fourier  transform  of  j (x,y) 
n n 


j (x,y)  = n^^  of  N subsections  of  blurred  Image 

n 


The  square-shaped  j overlap  each  other  halfway;  upper  right  and  lower  left  corners 
of  diagonally  adjacent  members  touch  each  other's  corners. 


Given  b(x,y),  Weiner  optimization  theory  suggests  that  the  filter  function 


H(p,q)  - 


|J(p.q)  l^|B(p.q)|^  + 


X = Arbitrary  positive  constant,  N “ Noise  spectral  density,  assumed 

white 


Validating  the  b-approxlmation: 


j^(x,y) 


ijj(x.y)  * b(x,y) 


l„(x,y> 


n^^  subsection  of  ideal  image 


N N 

I In  J (p,q)  « I In  I (p,q)  + N In  B(p,q) 


n»l 


n“l 


In (p,q) ,B(p,q)  ■ Fourier  transforms  of  1 (x,y),b(x,y) 

n 


N 

I In  I„(p,q) 
n«l 


N <ln  I > 
n 


constant 


average 


This  permits  an  expression  for  B(p,q)  in  terms  of  the  J^(p,q).  Inverse  transforma- 
tion then  gives  the  approximation  for  b(x,y)  in  the  text. 
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will  produce  deblurring.  As  before,  capitalized  quantities  represent  Fourier  trans- 
forms  of  their  lower  case  counterparts.  Constant  X establishes  the  tradeoff  between 
good  deblurring  on  one  hand  and  efficient  noise  rejection  on  the  other.  Physically, 
it  governs  filter  bandwidth. 

Haar  Implementations  of  b-identif icatlon  and  b-lnversion  are  presently  being  studied. 
The  b-identif icatlon  process,  which  requires  inverse  Fourier  transforms  of  products 
of  Nth  roots  faces  formidable  difficulties.  Other  approaches  may  have  to  be  con- 
sidered. The  b-inverslon  process,  however,  offers  more  opportunity  for  Haar 
innovation  since 


1/B  for  |j|  »N 


H = 


for  |J|^ 


> (N„/|J|  ) 


suggests  a piecewise  approach  to  implementation. 


5. 2. 2. 2. 2 Experimental  Procedure  - Experimental  tests  on  b-inversion  can  be  con- 
ducted even  while  the  overall  procedure  awaits  further  adaption  to  the  Haar  domain. 
These  tests  begin  with  an  Imperfect  image  j(x,y)  whose  blurring  is  due  to  a known 
b(x,y).  Examples  include  a pinhole  camera  photograph  whose  b(x,y)  follows  from  the 
numerical  value  of  the  hole  diameter.  Haar  expansions  of  b(x,y)  and  j(x,y)  can  be 
combined  in  various  ways,  using  the  above  piecewise  approximation  as  a rough  guide. 
Inverse  Haar  transformation  then  permits  direct  evaluation  of  the  result. 

One  way  of  combining  the  two  functions  to  achieve  possible  deblurring  would  take  the 
Haar  transform  of  b and  average  all  coefficients  of  a given  sequency  r.  Designating 
the  average  coefficient  as  a(r),  one  would  multiply  each  Haar  coefficient  of  j(x,y) 
by  either  l/a(r)  or  k a(r);  k “ constant.  The  choice  depends  upon  whether  the 


coefficient  exceeds  or  falls  below  some  present  threshold,  respectively.  Different 
threshold  settings  and  different  values  of  k can  be  tried. 

5. 2. 2. 3 Depth  of  Field  Blur  - Since  this  difficulty  nevertheless  permits  all 
various  parts  of  the  observed  field  to  be  brought  into  focus  with  different  lens 
settings,  a possible  solution  consists  in  recording  Haar  transforms  of  a given  scene 
with  several  different  lens  settings.  These  settings  should  cover  a range  over 
which  all  parts  of  the  scene  come  into  focus.  The  variation  of  any  particular  Haar 
coefficient  over  the  range  can  then  be  interpolated  from  the  recorded  samples,  and 
the  maximum  magnitude  evaluated.  This  maximum  represents  the  in-focus  value  of  the 
Haar  term  under  consideration.  Accordingly,  the  several  Haar  expansions  can  be 

4 

replaced  by  one  based  on  the  interpolated  maxima.  The  inverse  transform  should  then 
yield  an  image  simultaneously  in  focus  at  all  points.  In  practical  operation,  the 
required  multiple  lens  setting  may  be  achievable  via  the  same  dither  motion  used  in 
autofocus . 

5. 2.2.4  Jittered  Images  - 

5. 2. 2. 4.1  Concept  - Given  the  Haar  image  expansion 

i(x,y)  = nil  a(r,m,p,n)h(r,m,x)h(p,n,y) 
rmpn 

h(r,m,x)  » one-dimensional  Haar  function  of  sequency  r and  offset  m 

a(r,m,p,n)  = Haar  expansion  coefficient 
4 

analysis  shows  that  sinusoidal  jitter  variations  Ax  sin  wt  change  the  above  co- 
efficients to  time  varying  quantities: 

a(r,m+2*^  Ax  sin  wt,n,p) 


4 r/  2 “IT 

Because  h(r,m,x)  • 2 h(o,o,x-m.2  ),  x-directed  jitter  variations  can  be 

interpreted  as  equivalent  variations  in  m. 
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Image  jitter  can  therefore  be  corrected  In  the  Haar  domain  by  dithering  the  value  of 
m to  cancel  this  variation: 

Set  m * m-Am(t) 

m = constant 

Am(t)  = 2^Ax  sin  wt 

The  resulting  a(r,m,p,n)  can  be  Inverse-transformed  to  create  an  unjlttered  Image. 

The  above  remarks  have  simpli<^led  matters  by  treating  m as  a continuous  variable. 

Since  m is  in  fact  discrete,  one  approximate  approach  would  replace  Am(t)  by  the 
nearest  integer.  Other  approaches  based  on  interpolation  between  a-coef ficients 
with  successive  m remain  to  be  studied. 

These  remarks  assume  that  the  jitter  is  sinusoidal,  and  that  the  phase  and  amplitude 
can  be  detected.  The  sinusoidal  assumption  follows  from  the  fact  that  underdamped 
gyros  or  marginally  stable  servos  are  often  characterized  by  a single  critical 
resonance.  Phase  can  be  determined  from  the  periodic  nature  of  the  a-variation. 
Amplitude  determination  based  on  closed  loop  analogs  is  presently  under  study. 

5. 2. 2. A. 2 Experimental  Approach  - Experimental  verification  of  this  approach 
consists  in  jittering  test  patterns  sinusoidally,  using  a variety  of  amplitudes  and 
frequencies.  The  patterns  might  consist  of  vertical  bars  of  various  uniform  widths. 
These  widths  can  be  correlated  with  the  Haar  functions  that  seem  to  react  most  sen- 
sitively to  the  jitter.  For  convenience,  the  jitter  can  be  performed  quasi-statically 

5.2.3  Image  Enhancement 

5.2. 3.1  Objective  - Image  enhancement  covers  control  over  brightness,  contrast, 
edge  definition,  and  information  content.  As  in  autofocus,  manual  control  is  im- 
practical. The  eye-brain  combination  may  furthermore  fail  to  provide  an  adequate 
enhancement  criterion,  sacrificing  information  content  for  cosmetic  appearance.  Cur- 
rent plans  therefore  call  for  determining  the  Haar  sensitivity  to  control  brightness, 
contrast,  edge  definition,  and  information  content  for  the  enhancement  concepts  that 
follow. 
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5. 2. 3. 2 Brightness  Control 


5. 2. 3. 2.1  Concept  - Improper  brightness  level  can  lead  to  Inefficient  use  of 
available  gray  shade  levels  In  the  display.  Brightness  corresponds  to  the  coeffici- 
ent of  the  Haar  term 


h(-l,l,x)h(-l,l,y)  - h^^ 

when  the  expansion  Includes  the  entire  scene.  When  the  expansion  involves  only  a 
16x16  pel  block,  however,  the  coefficient  expresses  the  local  tone.  Under  these 
conditions,  brightness  corresponds  to  the  average  of  the  local  tone  over  all  16x16 
pel  blocks  within  the  scene. 

Automatic  brightness  control  consists  of  regulating  this  coefficient. 

One  approach  evaluates  the  average  coefficient  b^,  selects  a desired  brightness 

level  b„,  and  effectively  adds  b.-b,  to  t.,  the  h,  coefficient  of  1th  16x16  pel 
2 2 1 1 dc 

block.  Figure  5-5  outlines  the  process,  using  a linear  analog  of  the  two-dimension 
format. 

This  process  destroys  information  contained  in  the  brightness  level.  Should  this 

prove  significant,  an  attenuation  element  K within  the  AVG  block  will  produce  a 

brightness  level  of  b + Kb  , K “ 1-posltlve  constant,  which  can  possess  a moderate 
5 ^ ^ 

response  to  scene  changes  Ab^. 

5. 2. 3. 2. 2 Experimental  Procedure  - With  different  neutral  density  filters  in  front 
of  the  camera,  measurements  of  tj^,  t^,  t^,  ....  tj^,  •••!  and  b^  can  be 

made.  The  should  respond  to  filter  changes,  but  the  t^  should  not.  If  response 
time  measurements  are  not  considered  important,  the  portion  of  the  test  setup  between 
t^  and  tj^  can  be  computer  simulated. 


If  K> 


(rms  noise  voltage) /Abj^, 


brightness  information  will  generally  be  preserved. 
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Figure  5-5.  BRIGHTNESS  CONTROL  SCHEME 


5. 2. 3. 3 Contrast  Control  - 

5.2. 3.3.1  Concept  - As  In  brightness  control,  Improper  contrast  setting  leads  to 
inefficient  use  of  available  gray  shade  levels  in  the  display.  Contrast  and  activity 
represent  distinct  properties  of  a scene.  The  first  refers  to  i Lnimum  and  maximum 
gray  shade  levels  while  the  second  measures  mean  square  intensity  deviation.  Con- 
trast, furthermore,  does  not  possess  a straightforward  expression  in  the  Haar 
domain,  while  activity  does.  The  two  properties  nevertheless  possr'^s  a gross  simi- 
larity, since  they  tend  to  Increase  and  decrease  with  one  another. 
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A reasonable  solution  concept  might  therefore  consist  of  using  Haar-derived  activity 
as  a measure  of  contrast.  The  inverse  transform  from  the  Haar  domain  to  the  re- 
constructed image  would  then  use  this  measure  as  a means  of  contrast  control.  Ac- 
tivity can  be  conveniently  expressed  as  the  sum  of  squares  of  all  Haar  coefficients, 
excepting  the  h^^  term.  The  resulting  quantity  would  be  used  for  compensatory  gain 
control  in  the  inverse  transform. 

The  prime  counterexample  to  this  approach  is  a scene  which  largely  occupies  only  a 
few  gray  shades  about  the  neutral  level  but  which  nevertheless  contains  one  or  two 
hotspots  or  coldspots.  This  scene  would  possess  low  activity  and  high  contrast. 
Compensatory  control  would  respond  to  the  low  activity  by  producing  a high  gain, 
driving  hotspots  and  coldspots  into  saturation.  Information  penalties  should  be 
small,  however,  since  pictorial  details  would  not  be  lost. 

5. 2. 3. 3. 2 Experimental  Approach  - Calibrated  inputs  may  consist  of  different 

sample  scenes  representing  varying  degrees  of  contrast  about  a fixed  brightness 

level.  Alternatively,  inputs  can  be  achieved  just  by  turning  the  contrast  control 

knob  on  the  recording  camera.  Variations  in  Haar  transform  coefficients  in  response 

to  contrast  changes,  and  the  relative  constancy  of  the  h.  term,  can  then  be 

dc 

measured . 

t<ff-llne  processing  can  provide  the  compensatory  gain  control.  Contrast  level  in 
the  final  image  can  be  measured  by  means  of  peak-  and  minimum-voltage  detectors  at 
the  TV  display. 

5. 2. 3.4  Edge  Definition 

5. 2. 3.4.1  Concept  - Edge  definition,  or  enhancement,  can  clarify  the  outline  of 
potentially  important  shapes  within  the  perceived  image  field.  As  such  it  is 
closely  related  to  clutter  suppression. 

Within  the  Haar  transform  domain,  edge  and  texture  qualities  tend  to  overlap.  Tex- 
ture often  consists  of  fine  edge-like  details.  These,  however,  lack  the  continuity 
quality  of  true  edges,  extending  over  only  a few  pels.  Other  textural  aspects 
furthermore  correspond  to  spots  whose  Haar  properties  remain  distinct  from  edges. 
Thus,  the  overlap  between  edge  and  texture  properties  is  only  partial. 
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In  view  of  these  remarks,  simple  edge  enhancement  algorithms  will  include  some  tex- 
ture features.  Elimination  of  these  features  would  require  more  refined  algorithms 
of  presently  unknown  complexity.^ 

One  simple  algorithm  represents  edges  as  the  difference  between  focused  and  moder- 
ately defocused  Images.  In  the  Haar  domain,  the  defocused  image  cuts  off  all  se- 
quencies  above  relatively  restricted  limits.  Otherwise,  the  transforms  are  essen- 
tially the  same.  Edge-seeking  via  differencing  therefore  subtracts  the  two  trans- 
forms, leaving  nonzero  coefficients  only  in  a band  adjoining  the  high  sequency  bor- 
ders. Enhancement  amounts  to  adding  this  result  to  the  transform  of  the  original 
image.  Equivalently,  it  simply  amplifies  the  highest  sequency  Haar  functions  of  the 
original  image.  ' 

5. 2. 3. A. 2 Experimental  Approach  - Verification  follows  from  taking  the  Haar  trans- 
form of  superimposed  photographic  transparencies.  One  is  a high-contrast  positive, 
while  the  other  is  a lower-contrast,  moderately  blurred  negative  of  the  identical 
scene.  The  Inherent  subtraction  emphasizes  edges  by  suppressing  nonedges.  Results 
can  be  compared  with  a transform  of  the  positive  transparency  alone. 

5. 2. 3. 5 Texture  Definition  - Solution  concepts  and  experimental  procedure  follow 
from  the  preceding  discussion  of  edge  enhancement.  To  Include  sp  .c-llke  contribu- 
tions to  texture,  the  band  adjoining  the  high  sequency  borders  must  be  broadened. 

It  can,  in  fact,  include  all  sequencies  except  h^^  for  the  16x16  pel  block  under 
consideration. 

5.2. 3.6  Information  Content 


5. 2. 3. 6.1  Concept  - Although  maximum  bandwidth  is  usually  desired  to  utilize  the 
maximum  available  system  resolution,  information  content  of  a given  scene  can  be  de- 
graded by  choosing  a pass  band  that  is  too  high  for  the  MTF  of  the  imaging  system. 

The  excess  bandwidth  passes  excess  noise,  thereby  degrading  the  signal-to-nolse  ratio 
(SNR)  and  information  content: 


One  possible  approach  might  consist  of  comparing  In-focus  and  slightly  defocused 
Images,  to  see  if  the  change  suppresses  texture  while  retaining  major  edge  features. 
Another  algorithm  might  measure  edge  length,  rejecting  those  which  fall  below  a 
specified  limit.  Definite  recommendations,  however,  must  await  further  study. 


5-15 


l(blts) 
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Insufficient  bandwidth  also  degrades  I by  increasing  resolution  cell  size. 

Optimum  filter  theory  reconciles  these  conflicting  tendencies  by  choosing  an  algo- 
rithm which  minimizes  noise  subject  to  a given  transient  error. ^ 


5. 2. 3. 6. 2 Experimental  Approach  - In  line  with  footnote  7 below,  the  noise  floor 
of  the  Haar  transform  of  a given  image  would  be  evaluated.  This  corresponds  to  the 
asymptotic  level  that  squared  coefficients  appear  to  approach  at  the  highest  sequen- 
cles.  For  this  purpose,  the  Haar  transform  should  contain  sequencies  well  beyond 
the  highest  ones  present  in  the  image. 

The  squared  Haar  spectrum  will  sink  to  selected  multiples  of  this  level  at  certain 

sequencies.  Optimization  of  image  information  content  then  calls  for  using  only  that 

portion  of  the  Haar  domain  that  falls  below  these  sequencies  to  reconstruct  the 

2 

image.  Various  multiples  a should  be  selected.  Results  will  generally  show  that 
2 

lower  a give  crisper  but  noisier  pictures. 

5.2.4  Adaptive  Signal  Processing 

Specific  tasks  under  this  heading  Include  d-c  restoration  and  dynamic  range  compres- 
sion. D-c  restoration  covers  distortions  arising  from  a-c  coupling  of  FLIR  photo- 
detectors to  the  electronics.  Dynamic  range  compression  maximizes  utilization  of  the 
available  nu.nber  of  gray  levels  in  the  display. 


Transient  error  is  the  mean  square  deviation  between  noise-free  Images  before  and 
after  applying  the  algorithm.  Noise  refers  to  that  part  of  the  original  image 
noise  power  spectrum  N which  the  algorithm  falls  to  block.  In  Fourier  transform 
terminology,  the  algorithm  corresponds  to  filter  H and  the  original  image  to  signal 
G.  Variational  calculus  shows  that  H | Gl^/ ( | Gl^+a^N)  represents  the  desired 
filter.  Different  choices  of  Lagrange  multiplier  a2  yield  different  tradeoffs  be- 
tween transient  error  and  noise.  Physically,  a^  corresponds  to  bandwidth.  To  a 
first  order  approximation  this  filter  stipulates  unit  gain  for  all  spectral  regions 
exceeding  an  a^  multiple  of  the  noise.  Regions  below  this  level  are  completely 
blocked. 


5-16 


5. 2. 4.1  D-C  Restoration 


5. 2. 4. 1.1  Objectives  - Lack  of  d-c  restoration  in  the  FLIR  system  creates  three 

characteristic  difficulties:  Lost  horizon,  blacker-than-black  and  non-linear  opera- 

tion. Lost  horizon  might  arise  from  two  adjacent  horizontally  scanned  lines,  one 
above  the  horizon  (bright)  and  the  other  below  (dark).  In  passing  through  the  a-c 
amplifier,  the  respectively  high  and  low  d-c  pedestals  will  tend  to  decay  to  a common 

intermediate  gray  tone.  Cold  bars  limit  the  decay  by  effectively  re-imposlnt;  initial 

9 

conditions  at  the  beginning  of  each  successive  scan.  Drift  in  cold  bar  bias  level 
however  can  squeeze  all  available  a-c  borne  picture  Information  into  a few  display 
gray  levels,  leaving  the  others  unused.  Automatic  bias  adaption  would  therefore  fill 
a cogent  need. 

5. 2. 4. 1.2  Concept  - The  cold  bar  basically  Interjects  negative  rectangular  pulses 
at  points  where  one  scan  of  a given  line  ends  and  a new  one  begins.  These  pulses 
convert  a prolonged  d-c  pedestal  extending  over  many  scans  into  an  a-c  waveform. 

Short  single-scan  pedestals  counterbalance  the  negative  pulses  to  produce  a long- 
term zero  average  in  the  new  waveform.  Pedestal  shortening  relaxes  time  constant 
requirements  governing  a-c  amplifier  low-frequency  cutoff.  This  in  turn  reduces 
susceptibility  to  (l/f)-noise. 

Incorrect  pulse  amplitudes  however  can  force  the  pedestals  to  assume  inappropriate 
settings,  which  utilize  the  available  display  gray  levels  Inefficiently.  Haar  trans- 
form processing  can  relieve  this  difficulty.  It  intervenes  beween  sensor  and  dis- 
play, and  can  adjust  the  capacity  of  one  to  the  other.  Specific  corrective  schemes 
will  resemble  Haar  brightness  control,  discussed  earlier.  These  schemes  however 
cannot  impose  absolute  control,  which  would  negate  all  scene-brightness  changes, 
since  this  may  destroy  important  information.  Moderated  control  can  be  achieved  by 
proper  feedback  design  within  the  Haar-domain  portion  of  the  processor. 


Negative  step-signal  changes  from  this  incorrect  gray  tone  (rather  than  the  correct, 
brighter,  value)  plunge  to  voltage  levels  below  absolute  black;  i.e.,  blacker  than 
black.  Positive  steps  would  similarly  create  whiter  than  white.  If  there  are  too 
few  voltage  levels  to  accommodate  the  plunge,  saturation  sets  in. 

Decay  would  only  cause  severe  disability  If  continued  over  many  scans. 
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The  ability  of  the  cold  bar  to  minimize  transient  decay  Increases  with  scan  speed. 


10 

However,  a small  residual  of  the  decay  trend  will  always  remain,  tilting  the  bright 
pedestals  downward  (in  time),  and  dark  ones  upward,  toward  the  common  median  gray 
level.  Should  this  prove  objectionable,  additional  Haar  corrective  measures  can  be 
applied.  These  measures  would  recognize  that  within  any  sequency  Haar  functions  with 
larger  m suffer  greater  decay.  Haar  coefficients  for  larger  m can  accordingly  be 
amplified  a little  more  to  nullify  the  decay.  Amplification  can  vary  linearly  with 
m,  or  can  follow  some  simple  polynomial  law,  depending  upon  the  desired  completeness 
of  the  nullification. 

In  conjunction  with  the  preceding  work  on  d-c  restoration,  the  effects  of  scanning 
without  a cold  bar  (or  other  form  of  vertical  reference)  will  be  investigated. 

5. 2. 4. 1.3  Experimental  Procedure  - Digitized  images  will  be  input  to  the  IBM  370 
computer  and  their  scan  lines  will  be  subjected  to  various  amounts  of  a-c  coupling 
in  order  to  simulate  the  initial  conditions  of  the  d-c  restoration  problem.  Because 
of  the  one-dimensional  nature  of  the  problem,  experimental  verification  requires 
observation  of  the  pedestal  height  of  only  one  scan  line  at  a time.  Feedback  algo- 
rithms can  moderate  this  height  via  the  d-c  Haar  function.  Preliminary  study  sug- 
gests that  the  feedback  loop  gain  can  be  a relatively  low  bandwidth  function  with  a 
moderately  large  amplitude  level  at  dc.  Transient-tilt  correction  will  investigate 
similar  algorithms  operating  on  the  higher  sequency  Haar  functions. 

5. 2.  A. 2 Dynamic  Range  Compression  - 

5. 2.  A. 2.1  Objective  - Dynamic  range  compression  attempts  maximum  utilization  of 
the  available  display  gray  levels,  since  these  are  generally  quite  limited.  It  tends 
to  amplify  small  contrast  variations  at  the  cost  of  moderating  large  ones.  Usually, 
the  tradeoff  is  highly  advantageous  both  in  terms  of  cosmetic  appearance  and  infor- 
mation gain.  The  process  proposed  here  is  related  to  histogram  equalization,  but 
possesses  several  distinctive  features  which  suit  it  to  a Haar  implementation.  Haar 
implementation  in  turn  raises  the  possibility  of  real-time  operation. 


^*^Scan  speed  cannot  Increase  indefinitely,  since  the  upper  frequency  cutoff  limit  of 
the  photodetectors  would  ultimately  be  reached. 

^^Thls  assumes  unidirectional  rather  than  racetrack  scan.  With  slight  modification, 
the  above  corrective  procedure  can  accommodate  racetrack  scans  as  well. 
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5. 2. 4. 2. 2 Concept  - Histogram  equalization  regulates  the  small  signal  gain  at 

each  amplitude  in  proportion  to  the  number  of  pels  in  the  scene  which  occupy  that 

amplitude.  Gain  savings  achieved  in  the  low  occupancy  amplitudes  can  be  passed  on 

12 

to  the  high  occupancy  amplitudes. 

One  application  would  simply  apply  histogram  equalization  to  the  ensemble  of  h^^ 
from  all  the  16x16  pel  blocks  within  the  original  scene.  Picture  tone  would  then 
undergo  dynamic  range  equalization,  but  fine  intensity  variations  within  each  block 
would  remain  unaffected.  The  possibility  that  these  variations  may  also  require 
dynamic  range  compression  raises  a potential  difficulty.  This  difficulty  can  be 
addressed  through  a modified  version  of  the  general  scheme:  All  Haar  coefficients 

for  a given  sequency  r would  be  amplified  in  proportion  to  the  number  of  offset 
values  m that  are  occupied.  Occupancy,  here,  is  the  number  of  m whose  coefficients 
exceed  a preset  threshold  magnitude.  Designating  this  number  by  M(r) , the  gain  G(r) 
can  be  expressed  as 


G(r) 


: 

M(q)-2-q 

all  sequencles,  q 


Normalizing  term  2 


maximum  possible  value  of  M(r) ; i.e. , for  full  occupancy. 


The  ability  of  this  approach  to  reproduce  the  desirable  qualities  of  formal  histo- 
gram equalization,  and  possibly  to  improve  on  them,  is  currently  being  analyzed.  As 

in  formal  equalization,  a derivative  approach  that  treats  one  localized  region  at  a 
13 

time  will  be  considered. 


5. 2. 4. 2. 3 Experimental  Procedure  - Experimental  examination  of  the  G(r)  approach 
can  proceed  independently  from  the  theoretical  study  of  its  capabilities.  This 
examination  can  compute  G(r)  for  each  of  a number  of  scenes  and  observe  its  effect 


Total  variation  in  output  gray  level  ••  / Small  signal  gain  d Input  gray  level 

This  integral  cannot  exceed  an  allowed  upper  bound.  Since  small  signal  gain  must 
be  a monotonically  increasing  function,  lowered  values  of  the  integrand  in  certain 
intervals  of  the  input  gray  level  scale  permit  raised  values  in  other  regions 
without  exceeding  stated  bounds. 

13 

Termed  "Local  Area  Histogram  Equalization"  or  LAHE  in  the  literature. 
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on  the  reconstituted  image.  In  addition,  any  tendency  of  G(r)  to  assume  a repre- 
sentative form  applicable  to  a class  of  scenes  (e.g.,  ground  scenes)  can  be  evaluated. 

5.2.5  Data  Compression 

Data  compression  bears  a certain  resemblance  to  Dynamic  Range  Con^ression  (DRC)  but 
the  goals  are  different.  Here,  the  goal  consists  of  maximizing  information  content 
for  a limited  bit  capacity  rather  than  the  equally  likely  utilization  of  all  bit 
levels  (histogram  equalization). 

5. 2. 5.1  Objectives  - Data  compression  explores  the  means  for  acquiring  all  neces- 
sary picture  information  with  the  minimum  number  of  bits.  This  is  advantageous  where 
a data  link  is  utilized  to  transmit  FLIR  images  (in  real  time)  to  a remote  terminal. 

5. 2. 5. 2 Concept  - Three  different  approaches  are  being  considered.  One  approach 
simply  eliminates  all  Haar  terms  beyond  a present  sequency  limit.  The  second  elimi- 
nates all  terms  whose  coefficients  fall  below  a stated  threshold,  while  the  third 
approach  applies  thresholding  to  different  sequency  ranges. 

The  first  approach  is  clearly  the  simplest,  but  probably  the  least  efficient.  Fur- 
ther insight  may  be  gained  from  empirical  evaluation,  such  as  general  rules  for 
specifying  the  limit.  Thresholding,  as  used  in  the  second  approach,  should  prove 
more  effective.  This  approach  tends  to  discriminate  against  low-SNR  contributions 
to  image  data. 

Such  thresholding,  however,  can  conceivably  preserve  many  Haar  components  in  one 
sequency  range  while  destroying  all  components  in  another.  It  may  then  prove  that 
sacrificing  some  components  in  the  first  range  to  provide  bit  capacity  for  repre- 
senting a few  components  in  the  second  range  will  be  advantageous  despite  a net  loss 
in  SNR. 

The  third  approach  follows  this  reasoning.  It  subdivides  the  sequency  range  into 
low  and  high  zones  and  thresholds  each  one.  These  zones  in  turn  can  be  selected  to 
characterize  tone  and  edge  qualities. 
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5- 2. 5. 3 Experimental  Procedure  - The  three  approaches  can  be  tried  with  a variety 
of  scenes.  Qualitative  tradeoffs  between  simplicity  and  pictorial  quality  can  be 
noted.  Quantitatively,  experimental  measurements  can  determine  a best  setting  for 
the  present  sequency  limit  in  the  first  approach.  These  measurements  will  also 
determine  whether  this  limit  varies  markedly  from  scene  to  scene,  or  if  an  all- 
purpose limit  might  be  possible.  Similar  remarks  apply  to  threshold  settings  in  the 
second  and  third  approaches. 
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